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ABSTRACT 

Quantifying pilot visual behavior allows researchers to determine not only where a 
pilot is looking and when, but holds implications for specific behavioral tracking 
when these data are coupled with flight technical performance. Remote eye tracking 
systems have been integrated into simulators at NASA Langley with effectively no 
impact on the pilot environment. This paper discusses the installation and use of a 
remote eye tracking system. The data collection techniques from a complex human- 
in-the-loop (HITL) research experiment are discussed; especially, the data reduction 
algorithms and logic to transform raw eye tracking data into quantified visual 
behavior metrics, and analysis methods to interpret visual behavior. The findings 
suggest superior performance for Head-Up Display (HUD) and improved 
attentional behavior for Head-Down Display (HDD) implementations of Synthetic 
Vision System (SVS) technologies for low visibility terminal area operations. 
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1 INTRODUCTION 

Since its inception, NASA Langley Research Center (NASA LaRC) has conducted 
research in the field of Aeronautics, with an ever increasing interest in the 
human factors associated with aviation safety and the pilot/ flight deck 
interface. With this research scope, LaRC conducts research in flight deck 
interface design, employing various approaches to quantify pilot visual 
behavior (Comstock, Coates, & Kirby, 1985); (Spady & Waller, 1973). 



Today, NASA LaRC is working with state-of-the-art technology 
including the integration of remote eye tracking systems inside its commercial 
aviation simulators with minimal impact on the flight deck environment 
(Latorella and Ellis, 2010). An important aspect of eye tracking is to quantify 
visual behavior and to associate this behavior with quantitative metrics related 
to a pilot’s visual attention, cognitive processing, workload, and awareness or 
saliency of visual information (e.g., traffic awareness, prominence and 
importance of outside visual cues and references, effectiveness of visual alerts). 
Differences in pilot behavior across display variations can lead to significant 
findings in regard to the utility of various display types, location of display, 
complexity in usability, and operational comparisons in various flight 
environments. 

The development of quantitative behavioral metrics using eye tracking 
is explored by comparing eye tracking indicators to the behavioral indicators of 
display use and flight technical data. A review of the metrics developed for 
NASA’s NextSafe-2 experiment is presented to highlight the capabilities of 
NASA LaRC’s eye tracking installation and associated analytical methodologies. 

The NextSafe-2 experiment was a fixed-base simulation experiment 
investigating whether a lower decision height or reduced visibility minima is 
supported by the use of Synthetic Vision Systems (SVS). A SVS display was 
compared to a baseline blue-over-brown primary flight display in approach 
and landing operations (Ellis, Kramer, Shelton, Arthur III, & Prinzel, 2011). 
Along with flight technical analysis, eye tracking results were utilized, 
contrasting these display technologies with flight technical performance and 
human behavioral differences. The approach to oculometer hardware 
integration, methodologies for data reduction, and the impact of results are 
discussed. Further, this work points out future research needs to fulfill the 
development of quantitative behavioral metrics. 

1.2 Background 

Remote eye tracking systems first determine head position in all six degrees of 
freedom. This is done by two dimensional image recognition using several key 
facial characteristics. Points such as the eye corners, nostrils, corners of the 
mouth, ears, etc are identified and measured in relative pixel distance. 
Combining the located image points using two cameras of known position 
allows for 3D image processing, producing 6 degree of freedom head position 
values. Eye tracking is then measured by determining the center of the pupil 
through contrast image processing, relative to a glint reflection, provided by 
infra-red light sources of known location on the iris that indicates the center of 
the eye itself. By calculating the known distance between these two points, 
trigonometry is used to calculate a vector between the two points. A three 
dimensional eye gaze vector can be calculated in reference to a world 
coordinate system, such as a flight deck. A minimum of two cameras are 
required to perform 3 dimensional calculations (Ellis & Schnell, 2009). 



Reducing the data has historically been done with some scientific 
variance. Lookpoints from the gaze vector calculation are determined by the 
intersection point of the gaze vector with a predefined area of interest (AOI) 
where an AOI is within the subject’s visual environment, such as the primary 
flight display, or navigation display. Sub-AOIs may also be defined (e.g., the 
attitude indicator or airspeed read-out], Lookpoints may also be used in the 
determination of fixations and saccadic movement. There are no standard 
definitions of fixations and saccades, however, most research loosely defines a 
fixation as "a relatively stable eye-in-head position within some threshold of 
dispersion (~2 deg] over some minimum duration [200ms], and with a 
velocity threshold of 15-100 degrees per second” [Jacob & Karn, 2003], For our 
research, fixations are defined as spatial and time dependent, characterized by 
occupying a spatial radius of 2 degrees for a minimum of 200ms. Saccade 
movements are defined as the movement from one fixation to the next, 
measured in both angular distance and velocity. 


2 METHOD 

The Smart Eye remote eye tracking oculometers have been integrated into two 
of the simulation cabs in NASA Langley’s Cockpit Motion Facility. [Latorella, et 
al., 2010]describes the initial Integrated Flight Deck installation, which 
collected both crew members’ synchronized eye movement data. The 
following describes a similar installation in the Research Flight Deck [RFD] 
simulator for the left pilot seat. Each system is capable of using anywhere from 

2 to 8 cameras, outputting real-time eye gaze data across a 100 + degree visual 
tracking range. The Smart Eye system was selected due to its superior 
performance, coupled with the flexibility in the placement of cameras and 
illuminators. This flexibility was crucial for nearly seamless integration into the 
flight deck, minimizing intrusion to the flight deck environment, thus, 
maintaining a high level of simulator fidelity. 

2.2 Flight Deck Integration 

The cameras provided the desired level of eye gaze accuracy [approximately 2- 

3 degrees], without compromising simulation fidelity during the NextSafe-2 
experiment. Figure 1 shows the locations of the cameras and illuminators 
installed in the RFD simulator. Data was collected for the left seat pilot in the 
NextSafe-2 experiment; however, synchronized collection of a dual crew flight 
deck has been installed and successfully utilized in NASA Langley’s Integrated 
Flight Deck [IFD] for another HITL experiment investigating Data Comm 
interaction on the flight deck [Norman, et al., 2010], 

Validation and verification of system output was performed, assessing 
data collection quality and accuracy commensurate with previous experiments 
[Latorella, et al., 2010], The functional head box - the tracking region of the 
subject, centered on the designed eye point of the flight deck - was 



approximately one cubic foot Trade-offs in eye tracking camera placement and 
lens size were made to create a head box capable of capturing all pilot head 
movements (as dictated by the experiment tasks] yet maintain sufficient 
resolution for the image processing. 



Figure 1. Camera and Illuminator Locations in RFD 


2.3 Data Collection 

The raw data stream output of the two Smart Eye computers was recorded 
when the simulator was operating and was combined with simulator state 
data. The data were synchronized and time stamped for each trial. 

The collected eye tracking data for the NextSafe-2 experiment is 
shown in Table 1. AOIs were explicitly outlined by defining a world coordinate 
model of the interior flight deck and out-the-window (OTW) scene. The eye 
gaze vector information, especially when coupled with AOI information, 
provides explicit insight into which displays the pilots focused their attention, 
as well as visitation frequency and when the attention occurred. 


Table 1. Eye Tracker Output Parameters for NextSafe-2 


Frame Rate 

Eye Position X 

Frame Number 

Eye Position Y 

Head Position X 

Eye Position Z 

Head Position Y 

Gaze Direction A 

Head Position Z 

Gaze Direction B 

Head Position Quality 

Gaze Direction C 

Head Heading 

Gaze Heading 

Head Pitch 

Gaze Pitch 

Head Roll 

Gaze Direction Quality 

Time Stamp 

Pupil Diameter 

User Time Stamp 

Pupil Diameter Quality 

Closest World Intersection 



2.4 Data Reduction 


The data were reduced to support specific research objectives. 
Aviation research often uses specific windows of time or "snapshot data” (i.e., 
data collected at operationally significant points in time or space) to parse out 
the data and answer meaningful experimental questions and hypotheses. 
Specific to eye tracking applications, it is important to also define the data 
reduction parameters based upon the expected human behavior as demanded 
by the experiment tasks to elicit the best results. 

For the NextSafe-2 experiment, specified flight segments during the 
low visibility instrument approach and landing operations were defined, 
shown in Figure 2, and associated analyses were conducted based on 
assumptions of pilot visual behavior within these segments. Eye tracking data 
were used to support or refute the use of SVS to improve the safety and 
performance of instrument approach to landings. 

Within the "instrument segment,” the use of SVS was assumed to not 
significantly change the pilot’s visual behavior from that of conducting the 
instrument approach using the "blue-over-brown” baseline display condition. 
The only difference in the displays concepts was the presence of terrain and 
landing runway information on the PFD for the SVS concept. 

Around the 150 ft Height above threshold (HATh) point - the Decision 
Height (DH) for the instrument approach - the instrument-to-visual transition 
flight segment eye tracking data was analyzed to identify if SVS improved the 
pilot’s ability to visually transition from instrument to visual conditions. The 
contention was that SVS, because of its intuitive depiction of the runway, would 
induce a more efficient visual scan outside the airplane to find the runway and 
align the aircraft for landing. 

Within the Visual Segment, it was assumed that SVS would not affect 
visual behavior or attention. The pilot’s visual attention was assumed to be 
~100% directed out-the-window, attending to the pre-flare (line-up laterally 
and vertically) and preparing for the visual landing, including ‘clearing the 
runway’ or visually searching for the presence of other aircraft, vehicles, 
objects, or animals on the runway. 

Similarly, within the Flare Segment (flare to touchdown), it was 
assumed that SVS would not affect visual behavior or attention. The pilot’s 
visual attention was assumed to be 100% OTW, attending to the visual landing. 

Lastly, within the Landing Segment (touchdown to a safe taxi speed), it 
was assumed that SVS would not affect visual behavior or attention. The pilot’s 
visual attention was assumed to be 100% out-the-window, attending to the 
roll-out, tracking the runway centerline, and decelerating to a safe taxi speed. 

Based on these assumptions, percent of eye gaze (i.e., attention) OTW 
and number of transitions between OTW and head down was calculated for 
each analysis segment as a comparator across display conditions. 

Another inquiry to pilot behavior was the determination when pilots 
transitioned from instrument-flight to OTW (i.e., visual-flight) to complete the 
approach to land task. Determining the specific altitude, distance from 



threshold, and direction of the glance required new methods to analyze the 
data structured upon eye movement science. 

In development of the transition of attention algorithm, the AOI for 
which the lookpoint resided simplified the analysis. Time spent in an AOI by 
the lookpoint denoted attention, either head-up OTW or head-down on the 
instrument panel. 


150 ft HATh 



Landing Approach 

(50ft, "Flare" to Runway Exit) (lAFto 50 ft, "Flare") 

Figure 2. Eye Tracking Analysis Segments 

Eye tracking data was analyzed by creating a binary value to specify if 
the pilot attention was OTW or not. If the lookpoint was tagged in the OTW 
AOI, a value of 1 was assigned for that data point, if not, a zero was designated. 
If there was no calculated lookpoint due to loss of eye tracking, head tracking 
was utilized by referencing a running average of head tracking pitch. If the 
head pitch was greater than the head-up reference pitch, that data point was 
tagged as OTW, and a value of 1 was assigned. 

A moving time window of two (2) seconds, calculating an average of 
the past 2 seconds of the binary OTW value, was used to determine when the 
first occurrence of both a glance and a full transition OTW occurred. This ‘gated 
average’ value was then compared to a set-point threshold of 10% (200ms) to 
signify the occurrence of the "initial glance”, and 100% identifying a full 
transition to OTW, shown in Figure 3. The algorithm reports the reference 
times when the transition set points of 10% and 100% are reached, and 
simulator and eye tracking data such as altitude above field-level (AFL), 
distance from threshold, or gaze direction at that instant in time are recorded. 
The 200ms time duration was chosen as the initial glance threshold based 
upon 200 ms being the experimental definition of a single fixation. 

The 2-second time window was chosen due to the eye movement 
behavior of the pilots. Several size windows of time were reviewed to process 
the eye movement signal, including 10s, 8s, 6s, 4s, 2s, and Is, and evaluated 
over each pilot’s visual behavior to estimate when the defined transitions 
occurred. The time window size specifically impacts the full transition metric. 
An exceedingly large window neither quickly nor accurately captures a full 
transition point. In a approach and landing environment, small differences in 
time, (as small as 1 second of elapsed time) can result in large lateral and 
vertical distances. A window that is too small may result in an inaccurate 


depiction of a full transition, in which pilot attention is falsely classified as fully 
transitioned OTW when it may only be an extended glance between transitions. 
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Figure 3. Binary Attention and Signal Analysis 


2.5 Data Analysis 

The data were analyzed across the pilot subjects to determine if statistical 
significance existed in three operational comparisons; SVS display location 
(HUD vs. HDD), SVS HDD Equipage (SVS vs No SVS), and effects of single versus 
crewed pilot operations. The results presented herein specifically focus on eye 
tracking and its analysis. The operational results from the NextSafe-2 
experiment are discussed elsewhere (Kramer, et al., 2011). 

3 RESULTS 

The NextSafe-2 experiment observed a data collection rate of 87% across all 
experiment scenarios. Eye tracking was robust across all segments of the 
approach, with consistent and operationally rational results, providing strong 
operationally comparable human performance metrics. Spatial accuracy was 
observed in calibration of the eye tracking system for each subject to the order 
of approximately 2 to 3 degrees. The spatial accuracy proved sufficient due to 
the decisive pilot eye behavior in transitioning across gross AOIs on the HDDs 
and OTW. The spatial accuracy could have been improved but was not 
necessary to meet the research objectives. 


3.2 Flight Simulation Study Results 


The analysis of the eye tracking data was grouped into SVS operational 
comparisons, yielding several significant findings in SVS location and crewed 
versus single pilot operations. 

In Figure 4, differences in head-up percentage across eye tracking 
segments revealed that pilots retain a significant level of attention head down, 
even in "head-up” flight segments (i.e., ~30% during the visual segment of 
flight, and as much as 10% in the flare segment). This information reveals that 
pilots allocate a significant attention to guidance and instrumentation during 
these "visual” segments, suggesting this information is still utilized until the 
beginning of the flare. The HUD, with or without SVS, allows for this 
information to be observed while maintaining an OTW view, an impossible 
behavior to achieve with only HDD presentation of guidance and 
instrumentation. Transition count data between HDD and OTW, shown in 
Figure 4, provides insight into how pilots without a HUD transition their 
attention during the visual segment, indicating, on average, that two transitions 
occur (OTW to HDD, HDD to OTW). However, combining information from both 
transition count and head up percentage, the transition to head down accounts 
for 30% of the head down time, suggesting pilots are still flying guidance in the 
visual segment and not merely referencing it. 



Figure 4. SVS Operational Comparison SVS HUD vs. SVS HDD 

The specific altitude at which visual transition occurred was examined. The 
data showed that pilots using a HDD with SVS maintained their visual attention 
on the HDD until closer to the DH than without SVS. The data also showed that 
the time between a pilot’s initial and full transition to OTW. Table 2 shows the 
operational comparisons between baseline HDD and SVS HDD indicate the 
initial glance and full transition of attention, as defined previously. Transition 
of attention data also shows the difference in altitude for SVS operations are 
less than that without SVS. This finding, coupled with the findings of initial 
glance being closer to DH with SVS, suggest pilots operating with SVS are 
making a more decisive transition from HDD to OTW, thereby suggesting 
greater confidence in their trajectory and position and a more efficient visual 
behavior. 


Table 2. SVS Operational Comparison Visual Transition Altitude 



Baseline 

HDD 

SVS HDD 

Initial Glance AGL (ft) 

Mean 

367 

276 

Full Transition AGL (ft) 

Mean 

197 

178 


A comparison of utilization of SVS with crewed vs. single pilot 
operations was also conducted. This analysis was made to evaluate the 
influence of crew assistance on visual behavior and attention. Surprisingly, no 
significant differences in visual attention were found during the approach and 
flare segments. Significant findings were, however, observed during the 
landing/roll-out segment, shown in Figure 5, indicating single pilots made 
several more transitions between the HDD and OTW. This is explained by the 
difference in task loading between the two comparisons. Crewed operations 
allow for the pilot flying to maintain attention OTW with the other crew 
member providing speed, runway remaining, and turn-off information callouts 
especially using the NextSafe-2 advanced airport moving map display. These 
tasks are critical in the rollout phase, made particularly more difficult in low 
visibility operations, and are all tasked to the individual pilot in single crew 
operations. This information is only available head-down, requiring the single 
pilot to transition with increased frequency to retain critical attention OTW 
while at the same time collecting the necessary information from the HDDs. 



Figure 5. Crew of 2 vs. Single Pilot SVS Operational Comparison OTW/HDD 

Transition Count 


4 CONCLUDING REMARKS 

The eye tracking capabilities integrated into the flight simulators at NASA 
Langley provide a critical capability to HITL research. The measures of 
attention and transition of attention algorithm used in the NextSafe-2 


experiment demonstrate the significance and efficacy of characterizing pilot 
visual behavior. The data provide strong support to the efficacy of both HUD 
and SVS operations to improve the efficiency of pilot visual behavior and 
furthermore, provide a rationale to describe variability observed in flight 
technical performance. Without eye tracking these results would not have been 
discovered, and flight technical performance would be the sole comparator to 
define the operational advantages or disadvantages to SVS and are not typically 
significant. The methodologies produce quantifiable results across operational 
comparisons revealing visual behavior data affording researchers improved 
understanding of pilot performance. 
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